Tutorial: Least-Squares Fitting

Least-squares fitting, first developed by Carl Friedrich Gauss, is arguably the most widely used technique in statistical data analysis. It provides a method through which the parameters of a model can be optimised in order to obtain the best fit to a data set through the minimisation of the squared differences between the model and the data. This tutorial document describes the closely associated methods of least-squares and weighted least-squares (χ) fitting. We derive least-squares estimators both as Maximum-Likelihood (ML) estimators and as Best Linear Unbiased Estimators (BLUE), reconciling the two treatments in the conclusion. We then use the method to derive estimators for the parameters of some simple models, including the straight-line fit, together with the standard errors on the estimated parameters. We conclude with some general observations on how the lessons learned from the specific case of least-squares fitting can inform our understanding of machine vision and medical image analysis algorithms in general. 1 The Method of Least Squares The method of least squares has, at various times, been ascribed to a number of different authors, notably Gauss and Laplace. Plackett [?, ?] describes the historical development of the method, and was responsible for establishing that the fundamental results are due to Gauss. It is arguably the most commonly used statistical estimation procedure, being almost ubiquitous in science and engineering, and is usually one of the first to be learned. It provides a procedure through which a model can be fitted to a set of measurements by minimising the squared differences between the measurements and the model prediction,s with respect to the parameters of the model, in order to obtain the optimal parameters. However, this apparently simple yet powerful procedure depends on a set of assumptions that are much less well known than the method itself, leading to invalid applications. There are two, notably different, approaches to justifying the least-squares fitting procedure, differing in their assumptions. The (arguably) simpler and easier to interpret approach is to assume that the errors on the measurements are described by a normal distribution, in which case the least-squares estimators can be derived using maximum likelihood. However, it is also possible to derive least-squares estimators as those that, amongst all unbiased, linear estimators for linear models, have the lowest variance. We initially describe both derivations without comment, and reconcile them in the conclusion. 1.1 Derivation as a Maximum Likelihood Estimator Suppose that you have a set of n data (xi, yi) where i = 1...n, to which you wish to fit some model f(x). An implicit assumption is made at this stage that the model is a correct description of the physical process that generates the data (the consequences of using an incorrect model are described later). Noise will have been added to the data during the acquisition process, so that yi = f(xi) + ηi where ηi is the noise on the ith data point. Therefore, the residuals ri generated by subtracting the model prediction at xi from the measured yi are given by ri = yi − f(xi) = ηi We now need to apply this model to make statements regarding the degree of conformity of data. In the strictest sense, a distinction must be made between probabilities, which are defined over a range, and probability densities, which are not i.e. the probability P (ri|θ) that the residual ri will lie within the range r ± ∆/2 is given by